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ABSTRACT 

Interviews were conducted with contact persons in 
each of six Western states to assist in determining how the services 
and products of the Elementary and Secondary Education Act Title I 
Evaluation Technical Assistance Centers (TAC) could be used most 
effectively to help State and local education agencies improve the 
use and usefulness of evaluations. Respondents were asked to identify 
the biggest obstacle to the use of svaluatior data and how TAC could 
help to improve evaluation use* Data quality was the most frequently 
cited obstacle to effective use of evaluations. Other obstacles cited 
reference data interpretation, quality of objectives, timing of ^.he 
evaluation-report cycle, limitations of the Title I Evaluation and 
Reporting System, negative or inappropriate uses of data, confounding 
results across programs, focusing the evaluation reports, and 
training at the wrong levels More direct training on data 
interpretation and use and on implementation evaluation, assistance 
with local studies, and a refocusing of evaluation on the building 
level wer€ seen as vehicles for TAC to provide help" for evaluation 
imprcvement. (Author/HK) 
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^O^^^ VA^^^^^^^ HELP? 

•^^e B. Hansen 
^fitl^ I Evaluation 
hti^^^"^ Assistance Centers, 
TecP jtegio^s 9 and 10 

Educational Laboratory 

^hwest ^ 



THE PROat^^ 



How can s/"^*^^^ the Titl® I Evaluation Technical 

Assistanc^^ se^^j:s t^''5>rodu^^st Effectively to help SEAs and leas to 
improve t.^ Cen*^ ^ used evaluations at their respective levels. 




j,^^ J 0^^^ in meeting the evaluation requirements 

^<=> a^^i^ a"'\j,on implementing the Title I Evaluation 

2. ^ ^ c-A^ ^^''Pcove their Title I programs through the 

be^l^^i^^ ^r,<i^J' ti^^ evaluation ctices- 

TAG ^ - 
^^^^iffS^ 

t^ee 5^^^^^ UstJ 

APPROACH .V. ^ f^\t^ 

• Y ^ *^^v ^ti<^*i-- observations, discussions, field 
*^^t:eS. 

• to ^^^^ .40^ — evaluation questionnaires, surveys, 
.^rroaJ-.^w^ ^oliec^* 

Info rm ^ i.^ -.aUO"^ Part of the process of providing 

*^3pt /^-"'^^v *^^sed on informal data or experience . 

i*^'-- , ^at^ i^jn includes: 

^rmaj: Collect* 

I, * (l^^S) . Large scale-comprehensive survey of 

^*AC t:^ STOD^ 0ee<as for service. Data used for assisting 

^^i€nt^^ec:;i^ ^sses^ to ©inphasi^e; how to allocate resources in 
Ac if / t^^^^-tig wH3 to gain greatest effect. 
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-^^hv^ ,*f ^^^^ (1980-81) . More limited -focused" 
^'^AL ^/'s^j^^^^viE'f *;ervi^^,g ^,itb each state contact person in 
j^Vady ^^fi' of ^ j,es included due toj (1) logistics, and 

g^Hg^Lnot^n^plete. 



TAG LOCAL EVALUATION USE INTERVIEW STUDY 



QUESTIONS: 

1. Wbatr in your viewr is the biggest obstacle to effective use of 
evaluation data at the SEA level and the LEA level? 

2. How can TAG help your SEA to improve uses of evaluation 

• at the SEA level? 

• at the LEA level? 
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Does improving the uses of evaluation require greater TAG involvement 
with LEAS? (Asked only in states where direct contact between TAG 
and LEA is permitted.) 



RESPONSES 
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biggest Obstacle: 



r-nreting results 

People don't know what the data meansr especially NCEs. 

j^^tjievement test data has limited value, doesn't give full 
• picture of programr therefore can't have much value in 
pro9^^^ improvement decisions. 

ing objectives, relating outcomes to objectives* how 

b) Wri ^y^^^^mation relative to differences be used in making 

ram improvement decisions? Closer link needed between data 
P^^^p^Qgram planning. Objectives should provide the link. 

*^ow can 

- cu^e, but program implementation-evaluation may help. 
>^ot really 
^me exampi^^' 

✓vice on program implementation evaluation. 
% inse^^^-^ 

^xjice on materials implementation evaluation. 
% inse^^^-^ 

K Many I'EAs are not motivated to use evaluation in any way 

Problem: compliance fashion, 

^tber thar^ a 

^' ^ invol^^^"^^^^ 

J in this state because of state policy.) 




stacle: 

gjjgjity. Data are only useful to the extent that data 
15^^^, ^.y is high. Don't want LEAs to over-interpret gains when 
^"^^■*tv is suspect. 

-g emphasized this. Data quality is improving each year. 

improves, it becomes laore useful. 
AS it ■^»^ 



2. How Can TAG Help: 



Help find an alternate solution to regression effect error 
probleiHr something other than separation of pretest and 
selection. 

New evaluation regulations have provided SEAs-LEAs motivation to 
use results. An example is the sustained effects requirement. 
TAG can help by helping LEAs to learn how to ask themselves the 
question "What do I really want to know from these data and how 
can I find the answer?" 

Instruction on data use. 



3. LEA Involvement 



LEAs who want more TAG service are those with problems (gains 
are too low). Those satisfied with their gains don't want as 
much help. They are not as concerned abput data use . There are 
two kinds of interest in data use ; 

1. Statistically oriented person is interested in problems 
with the data . 

2. Instruct ionally oriented person is interested in problems 
indicated by the data. 

This implies need for different kinds of technical assistance. 



State No. 3 

1. Biggest obstacle: 

Depends on definition of "use." 

a. One definition implies that evaluation results can point to 
problems with Title I projects at individual grades within a 
school. 

Use, then is contingent on sufficient data at each grade. Rough 
generalizations can then be made on effectiveness and related to 
project characteristics. 

Problems that inhibit use under this definition: 

• Improper implementation of evaluation models resulting in 
invalid data, can lead to invalid decisions or recognition 
that data are invalid may lead to non-use. 
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Another definition of "use." 



Being able to provide information that goes into a dependent 
variable in a quasi-experimental study of different program 
characteristics. 

Problem-constraints to doing experimental studies in the fields 
e«g«, control of variables, random assignment, etc. 

2. How Can TAG Help: 

a. TAG can help make sure models are properly implemented, through 
LEA-TAG communication. TAG can help LEAs be more aware of 
dangers of interpretations from invalid data. 

b. TAG can help LEAs identify designs that are valid ways of 
answering locally generated questions. This includes specifying 
additional data beyond TIERS; specifying ways of defining 
variables of interest that can be studied; helping with design 
o-f analyses of data; helping to identify constraints that would 
lead to invalid interpretations. 



3. LEA Involvement: 

Depends on LEA and evaluation use definition as indicated. 



State No. 4 

1. Biggest Obstacle: 

Some districts are using negative results (poor gains) to shake up 
the i r progr ams . 

Positive results aren't being used as much (as they could/should be.) 



2. How Can TAG Help: 

• Workshops on interpretations of results. 

• Teaching LEAs to key their decisions to certain partii of 
reports, (presumably, those parts which are most pertinent to 
the decision being made.) 



3. LEA Involvement 

(Not asked due to logistic and policy reasons*) 
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state No, 5 



1. Biggest Obstacle: 

• Unreliability of data. Getting reliable data requires doing 
something we don*t know how to do. 

• Reporting cycle-timing is off for effective state level use of 
data. 

• Confounding of results across programs. A child may be 
"impacted" by multiple programs with each program claiming 
crediw for positive results. 

• Wrong information is collected for LEA use. 

• It never occurs to LEA staff that what they are doing might not 
be effective. They focus on individual outcomes more than group. 



2. How Can TAC Help: 



• Little can be done to influence use of TIERS data. TIERS is not 
organized for use at LEA level. 

• Longitudinal data might help. 

• TAC has influenced better knowledge and use of tests and 
measurement. 



State No. 6 



1. Biggest Obstacle: 

e Emphasis on models and pre and post-testing detracts from other 
kinds of data collection and use. To many^ LEAs pre-post 
testing i^ evaluation. 

• Testing-planning cycle. Posttest data come in after the nex'c 
year's plan is in. This could be alleviated by mid-year testing 
but norm dates are incompatible. 

• Teachers look at year to year gain. They are not used to caking 
a longitudinal view. Also, practical constraints such as 
student mobility affect longitudinal view. 

• Level at which evaluation has focused, e.g., administrator is 
not appropriate. There should be more focus on the classroom ^ 
teacher level. The lack of awareness of potential benefits of » 
evaluation begins the classroom teacher. 
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How Can TAG Help: 

• Provide more training on sustained and implementation evaluation. 

• Focus more on building level personnel; teachers and principals. 

LEA Involvment. 
(Not asked.) 



SUMMARY 

Interviews were conducted with the TAC contact person in each of six 
western states to obtain their views on two major questions pertaining to 
the use of evaluation data. 

1) What is the biggest obstacle to the use of evaluation data at 
the SEA and the LEA levels? 

2) How can the TAC help to improve the use of evaluation? 

Data quality was the most frequently cited obstacle to effective use of 
evaluation data^ with three of the respondents referring to this issue in 
seme form. Other responses included: data interpretation, quality of 
objectives, timing of the evaluation-reporting cycle, limitations of 
TIERS, negative or inappropriate use of data, confounding of results 
across programs, focusing the evaluation reports and training at the 
wrong level. 

As can be seen, the responses to this question varied widely, 
indicating a variety of problems rather than any clearly defined core of 
common concerns. To some extent this reflects the differences in 
conditions within states, and to some extent it reflects differences in 
interests, skills and philosoE^ies of the interviewees. For example, in 
one state where data quality is of major importance, a considerable 
effort has been mounted by the state to improve the quality of data 
through centralized data collection and use of a computer to reduce 
errors in computation. This state was one of the first to implement the 
TIBKS computer program and has kept abreast of all changes and 
iix^rovements in the use of computers for TIERS analysis. 

Alternative definitions of evaluation use were seen as important in a 
state wherein the contact person was a professional evaluator* This 




erson*s interest in pursuing research questions at the local level was 
also reflected in the obstacles cited. 

In general, the obstacles to evaluation use were somewhat 
idiosyncratic by state, however. 

On the question of how TAC might help Improve evaluation use at the 
LEA and SEA level, responses were less varied. More direct training on 
data interpret ion and use and on implementation evaluation were seen as 
helpful. Assistance with local studies and a refocusing of evaluation on 
tt Iding level were also seen as helpful. 

These results are admittedly sketchy and are based on interviews with 
only six state contacts. During the next two to three mont^.^i, similar 
interviews will be conducted with eight more states. The results of all 
fourteen interviews will be used to analvre TAC services and goals and if 
implications for change nppc-r, new service priorities will be 
established. It woulc w- pii:mature to base any conclusions on these 
early results but some obf^ervations and possible implications may be 
worth a few comments. 

Qfcccrvations and Possible Implications 
1. may be as ir.portant to note certain omissions from the 

responses as it is to note the responses themselves. For example, in an 
attempt to become more cost efficient and to create a legacy of 
instructional and reference materials which will outlast the TACs 
themselves, the Title-I TACs nationally are engaged in a coordinated 
effort to produce high quality instructional materials aimed at specific 
evaluation problems, ^ile this effort has been underway for more than a 
year, none of the state contact persons mentioned instructional materials 
either in their problem or their solution statements. 
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2. It appears that there is still some frustration with perceived 
deficiencies in the TIERS system ana perhaps with the inherent usefulness 
of aggregated achievement data to program planners and iinplementers. 

3, TAG cervices seem to be perceived as useful in general and as 
having at least potential value in solving some of the problems related 
to evaluation use. 
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